Remote Heart Rate Estimation Based on Transformer with Multi-Skip Connection Decoder: Method and Evaluation in the Wild.

Heart rate is an essential vital sign to evaluate human health. Remote heart monitoring using cheaply available devices has become a necessity in the twenty-first century to prevent any unfortunate situation caused by the hectic pace of life. In this paper, we propose a new method based on the transformer architecture with a multi-skip connection biLSTM decoder to estimate heart rate remotely from videos. Our method is based on the skin color variation caused by the change in blood volume in its surface. The presented heart rate estimation framework consists of three main steps: (1) the segmentation of the facial region of interest (ROI) based on the landmarks obtained by 3DDFA; (2) the extraction of the spatial and global features; and (3) the estimation of the heart rate value from the obtained features based on the proposed method. This paper investigates which feature extractor performs better by captioning the change in skin color related to the heart rate as well as the optimal number of frames needed to achieve better accuracy. Experiments were conducted using two publicly available datasets (LGI-PPGI and Vision for Vitals) and our own in-the-wild dataset (12 videos collected by four drivers). The experiments showed that our approach achieved better results than the previously published methods, making it the new state of the art on these datasets.

A Machine Learning-Based Correlation Analysis between Driver Behaviour and Vital Signs: Approach and Case Study.

Driving behaviour analysis has drawn much attention in recent years due to the dramatic increase in the number of traffic accidents and casualties, and based on many studies, there is a relationship between the driving environment or behaviour and the driver's state. To the best of our knowledge, these studies mostly investigate relationships between one vital sign and the driving circumstances either inside or outside the cabin. Hence, our paper provides an analysis of the correlation between the driver state (vital signs, eye state, and head pose) and both the vehicle maneuver actions (caused by the driver) and external events (carried out by other vehicles or pedestrians), including the proximity to other vehicles. Our methodology employs several models developed in our previous work to estimate respiratory rate, heart rate, blood pressure, oxygen saturation, head pose, eye state from in-cabin videos, and the distance to the nearest vehicle from out-cabin videos. Additionally, new models have been developed using Convolutional Neural Network (CNN) and Bidirectional Long Short-Term Memory (BiLSTM) to classify the external events from out-cabin videos, as well as a Decision Tree classifier to detect the driver's maneuver using accelerometer and gyroscope sensor data. The dataset used includes synchronized in-cabin/out-cabin videos and sensor data, allowing for the estimation of the driver state, proximity to other vehicles and detection of external events, and driver maneuvers. Therefore, the correlation matrix was calculated between all variables to be analysed. The results indicate that there is a weak correlation connecting both the maneuver action and the overtaking external event on one side and the heart rate and the blood pressure (systolic and diastolic) on the other side. In addition, the findings suggest a correlation between the yaw angle of the head and the overtaking event and a negative correlation between the systolic blood pressure and the distance to the nearest vehicle. Our findings align with our initial hypotheses, particularly concerning the impact of performing a maneuver or experiencing a cautious event, such as overtaking, on heart rate and blood pressure due to the agitation and tension resulting from such events. These results can be the key to implementing a sophisticated safety system aimed at maintaining the driver's stable state when aggressive external events or maneuvers occur.

Neural Network Model Combination for Video-Based Blood Pressure Estimation: New Approach and Evaluation.

One of the most effective vital signs of health conditions is blood pressure. It has such an impact that changes your state from completely relaxed to extremely unpleasant, which makes the task of blood pressure monitoring a main procedure that almost everyone undergoes whenever there is something wrong or suspicious with his/her health condition. The most popular and accurate ways to measure blood pressure are cuff-based, inconvenient, and pricey, but on the bright side, many experimental studies prove that changes in the color intensities of the RGB channels represent variation in the blood that flows beneath the skin, which is strongly related to blood pressure; hence, we present a novel approach to blood pressure estimation based on the analysis of human face video using hybrid deep learning models. We deeply analyzed proposed approaches and methods to develop combinations of state-of-the-art models that were validated by their testing results on the Vision for Vitals (V4V) dataset compared to the performance of other available proposed models. Additionally, we came up with a new metric to evaluate the performance of our models using Pearson's correlation coefficient between the predicted blood pressure of the subjects and their respiratory rate at each minute, which is provided by our own dataset that includes 60 videos of operators working on personal computers for almost 20 min in each video. Our method provides a cuff-less, fast, and comfortable way to estimate blood pressure with no need for any equipment except the camera of your smartphone.

Threats Detection during Human-Computer Interaction in Driver Monitoring Systems.

This paper presents an approach and a case study for threat detection during human-computer interaction, using the example of driver-vehicle interaction. We analyzed a driver monitoring system and identified two types of users: the driver and the operator. The proposed approach detects possible threats for the driver. We present a method for threat detection during human-system interactions that generalizes potential threats, as well as approaches for their detection. The originality of the method is that we frame the problem of threat detection in a holistic way: we build on the driver-ITS system analysis and generalize existing methods for driver state analysis into a threat detection method covering the identified threats. The developed reference model of the operator-computer interaction interface shows how the driver monitoring process is organized, and what information can be processed automatically, and what information related to the driver behavior has to be processed manually. In addition, the interface reference model includes mechanisms for operator behavior monitoring. We present experiments that included 14 drivers, as a case study. The experiments illustrated how the operator monitors and processes the information from the driver monitoring system. Based on the case study, we clarified that when the driver monitoring system detected the threats in the cabin and notified drivers about them, the number of threats was significantly decreased.

Estimation of Motion and Respiratory Characteristics during the Meditation Practice Based on Video Analysis.

Meditation practice is mental health training. It helps people to reduce stress and suppress negative thoughts. In this paper, we propose a camera-based meditation evaluation system, that helps meditators to improve their performance. We rely on two main criteria to measure the focus: the breathing characteristics (respiratory rate, breathing rhythmicity and stability), and the body movement. We introduce a contactless sensor to measure the respiratory rate based on a smartphone camera by detecting the chest keypoint at each frame, using an optical flow based algorithm to calculate the displacement between frames, filtering and de-noising the chest movement signal, and calculating the number of real peaks in this signal. We also present an approach to detecting the movement of different body parts (head, thorax, shoulders, elbows, wrists, stomach and knees). We have collected a non-annotated dataset for meditation practice videos consists of ninety videos and the annotated dataset consists of eight videos. The non-annotated dataset was categorized into beginner and professional meditators and was used for the development of the algorithm and for tuning the parameters. The annotated dataset was used for evaluation and showed that human activity during meditation practice could be correctly estimated by the presented approach and that the mean absolute error for the respiratory rate is around 1.75 BPM, which can be considered tolerable for the meditation application.

In-Vehicle Situation Monitoring for Potential Threats Detection Based on Smartphone Sensors.

This paper presents an analysis of modern research related to potential threats in a vehicle cabin, which is based on situation monitoring during vehicle control and the interaction of the driver with intelligent transportation systems (ITS). In the modern world, such systems enable the detection of potentially dangerous situations on the road, reducing accident probability. However, at the same time, such systems increase vulnerabilities in vehicles and can be sources of different threats. In this paper, we consider the primary information flows between the driver, vehicle, and infrastructure in modern ITS, and identify possible threats related to these entities. We define threat classes related to vehicle control and discuss which of them can be detected by smartphone sensors. We present a case study that supports our findings and shows the main use cases for threat identification using smartphone sensors: Drowsiness, distraction, unfastened belt, eating, drinking, and smartphone use.